2.1 How Lexical Tokens Match

A lexical rule must be used to define how to match a lexical token.

For instance:

%keyword FOO "foo"

Means that FOO is a reserved language keyword, matched as such by looking up into a keyword table, See (grammar-fw)keyword Decl. This is because "foo" will be converted to FOO in the lexical analysis stage. Thus the symbol FOO won’t be available any other way.

If we specify our token in this way:

%token <symbol> FOO "foo"

then FOO will match the string "foo" explicitly, but it won’t do so at the lexical level, allowing use of the text "foo" in other forms of regular expressions.

In that case, FOO is a symbol-type token. To match, a symbol must first be encountered, and then it must string-match "foo".

Caution:

Be especially careful to remember that "foo", and more generally the %token’s match-value string, is a regular expression!

Non symbol tokens are also allowed. For example:

%token <punctuation> PERIOD "[.]"

filename : symbol PERIOD symbol
         ;

PERIOD is a punctuation-type token that will explicitly match one period when used in the above rule.

Please Note:

symbol, punctuation, etc., are predefined lexical token types, based on the syntax class-character associations currently in effect.